Radar-based Gesture Recognition Research Articles

A mmWave MIMO Radar-Based Gesture Recognition Using Fusion of Range, Velocity, and Angular Information

A mmWave MIMO Radar-Based Gesture Recognition Using Fusion of Range, Velocity, and Angular Information

TRANS-CNN-Based Gesture Recognition for mmWave Radar.

In order to improve the real-time performance of gesture recognition by a micro-Doppler map of mmWave radar, the point cloud based gesture recognition for mmWave radar is proposed in this paper. Two steps are carried out for mmWave radar-based gesture recognition. The first step is to estimate the point cloud of the gestures by 3D-FFT and the peak grouping. The second step is to train the TRANS-CNN model by combining the multi-head self-attention and the 1D-convolutional network so as to extract the features in the point cloud data at a deeper level to categorize the gestures. In the experiments, TI mmWave radar sensor IWR1642 is used as a benchmark to evaluate the feasibility of the proposed approach. The results show that the accuracy of the gesture recognition reaches 98.5%. In order to prove the effectiveness of our approach, a simply 2Tx2Rx radar sensor is developed in our lab, and the accuracy of recognition reaches 97.1%. The results show that our proposed gesture recognition approach achieves the best performance in real time with limited training data in comparison with the existing methods.

RadarSense: Accurate Recognition of Mid-air Hand Gestures with Radar Sensing and Few Training Examples

Microwave radars bring many benefits to mid-air gesture sensing due to their large field of view and independence from environmental conditions, such as ambient light and occlusion. However, radar signals are highly dimensional and usually require complex deep learning approaches. To understand this landscape, we report results from a systematic literature review of ( N =118) scientific papers on radar sensing, unveiling a large variety of radar technology of different operating frequencies and bandwidths and antenna configurations but also various gesture recognition techniques. Although highly accurate, these techniques require a large amount of training data that depend on the type of radar. Therefore, the training results cannot be easily transferred to other radars. To address this aspect, we introduce a new gesture recognition pipeline that implements advanced full-wave electromagnetic modeling and inversion to retrieve physical characteristics of gestures that are radar independent, i.e., independent of the source, antennas, and radar-hand interactions. Inversion of radar signals further reduces the size of the dataset by several orders of magnitude, while preserving the essential information. This approach is compatible with conventional gesture recognizers, such as those based on template matching, which only need a few training examples to deliver high recognition accuracy rates. To evaluate our gesture recognition pipeline, we conducted user-dependent and user-independent evaluations on a dataset of 16 gesture types collected with the Walabot, a low-cost off-the-shelf array radar. We contrast these results with those obtained for the same gesture types collected with an ultra-wideband radar made of a vector network analyzer with a single horn antenna and with a computer vision sensor, respectively. Based on our findings, we suggest some design implications to support future development in radar-based gesture recognition.

Improving the Accuracy of Spiking Neural Networks for Radar Gesture Recognition Through Preprocessing.

Event-based neural networks are currently being explored as efficient solutions for performing AI tasks at the extreme edge. To fully exploit their potential, event-based neural networks coupled to adequate preprocessing must be investigated. Within this context, we demonstrate a 4-b-weight spiking neural network (SNN) for radar gesture recognition, achieving a state-of-the-art 93% accuracy within only four processing time steps while using only one convolutional layer and two fully connected layers. This solution consumes very little energy and area if implemented in event-based hardware, which makes it suited for embedded extreme-edge applications. In addition, we demonstrate the importance of signal preprocessing for achieving this high recognition accuracy in SNNs compared to deep neural networks (DNNs) with the same network topology and training strategy. We show that efficient preprocessing prior to the neural network is drastically more important for SNNs compared to DNNs. We also demonstrate, for the first time, that the preprocessing parameters can affect SNNs and DNNs in antagonistic ways, prohibiting the generalization of conclusions drawn from DNN design to SNNs. We demonstrate our findings by comparing the gesture recognition accuracy achieved with our SNN to a DNN with the same architecture and similar training. Unlike previously proposed neural networks for radar processing, this work enables ultralow-power radar-based gesture recognition for extreme-edge devices.

Radar-Based Gesture Recognition Under Ego-Motion for Automotive Applications

Radar-Based Gesture Recognition Under Ego-Motion for Automotive Applications

PointNet+LSTM for Target List-Based Gesture Recognition With Incoherent Radar Networks

Radar-based gesture recognition can provide autonomous electronic systems with a reliable way to infer a human's intention, e.g., in traffic environments involving vulnerable road users. Particularly in complex scenarios, algorithms operating on radar target lists derived from constant false-alarm rate outputs present an attractive solution, as they not only enable the filtering of relevant targets, but can also make full use of the diverse, high-resolution target parameters provided by modern radar sensors. Therefore, this article proposes PointNet+long short-term memory (LSTM) for the enhanced target list-based recognition of challenging traffic gestures, combining per-frame feature extraction with PointNet and learning from sequences with a LSTM. The approach is generalized to facilitate the use of multistatic radar data from sensor networks to exploit slightly different viewing angles, which is particularly helpful for motions with low radial velocity. The proposed method is validated on a comprehensive dataset comprising eight traffic gestures and data recorded from 35 participants. Measurements are conducted both indoors and outdoors with an incoherent radar sensor network comprising three chirp sequence–multiple-input multiple-output sensors. On this challenging dataset, our approach clearly outperforms a reference convolutional neural network, reaching up to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$92.2 \%$</tex-math></inline-formula> cross-validation accuracy.

Learning on Multistatic Simulation Data for Radar-Based Automotive Gesture Recognition

Radar-based gesture recognition can play a vital role in autonomous vehicles’ interaction with vulnerable road users (VRUs). However, in automotive scenarios the same gesture produces strongly differing radar responses owing to the wide range of variations such as position, orientation, or ego-motion. Since including all kinds of modifications in a measured dataset is laborious, gesture simulations alleviate the measurement effort and increase the robustness against edge and corner cases. Hence, this article presents a flexible geometric human target model allowing the direct introduction of a wide range of modifications, while it facilitates the handling of shadowing effects and multiradar constellations. Using the proposed simulation model, a dataset recorded with a radar sensor network consisting of three chirp sequence (CS) radars is resimulated based on the motion data simultaneously captured with a stereo video system. Completely substituting the measured by the simulated data for training, a convolutional neural network (CNN) classifier still achieves 80.4% cross-validation accuracy on a challenging gesture set, compared with 89.4% for training on the measured data. Moreover, using the simulated data the classifier is shown to successfully generalize to new scenarios not observed in measurements.

Radar-Based Gesture Recognition Using a Variational Autoencoder With Deep Statistical Metric Learning

Radar-based gesture recognition systems are a promising concept for new human–machine interfaces. However, the systems are sensitive to user-dependent gesture patterns, sensor noise characteristics, background environments, and unknown gestures or background motions. To overcome such challenges, we propose a variational autoencoder architecture-based deep metric learning, which is optimized using a novel loss function combining a statistical distance triplet loss and the center loss. By learning the statistical distance between distributions, the proposed solution, compared with triplet loss, is able to better learn the nonlinear characteristics in the data, is less sensitivity to training strategy, and is capable of creating close knit embedding clusters. Moreover, it is experimentally demonstrated that the proposed gesture recognition system using radar spectrograms has improved classification accuracy and random gesture rejection accuracy compared with the state-of-the-art deep metric learning approaches. Furthermore, we proof the real-world capability of the system by implementing a real-time demonstrator, which allows playing “Tetris” with hand gestures.

Spiking Neural Network-Based Radar Gesture Recognition System Using Raw ADC Data

One of the main challenges in developing embedded radar-based gesture recognition systems is the requirement of energy efficiency. To facilitate this, we present an embedded gesture recognition system using a 60 GHz frequency modulated continuous wave radar using spiking neural networks (SNNs) applied directly to raw analog-to-digital converter (ADC) data. The SNNs are sparse in time and space, and event driven, which makes them energy efficient. In contrast to the previous state-of-the-art methods, the proposed system is only based on the raw ADC data of the target, thus avoiding the overhead of performing the slow-time and fast-time Fourier transforms (FFTs). Furthermore, the preprocessing slow-time FFT is mimicked in the proposed SNN architecture, where the proposed model processing speed of 112 ms advances the state-of-the-art methods by a factor of more than 2. The experimental results demonstrate that despite the simplification, the proposed implementation achieves recognition accuracy of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$98.1 \%$</tex-math></inline-formula> , which is comparable with the conventional approaches.

Dop-DenseNet: Densely Convolutional Neural Network-Based Gesture Recognition Using a Micro-Doppler Radar

Hand gesture recognition is an efficient and practical solution for the non-contact human–machine interaction in smart devices. To date, vision-based methods are widely used in this research area, but they are susceptible to light conditions. To address this issue, radar-based gesture recognition using micro-Doppler signatures can be applied as an alternative. Accordingly, the use of a novel densely convolutional neural network model, Dop-DenseNet, is proposed in this paper for improving hand gesture recognition in terms of classification accuracy and latency. The model was designed with cross or skip connections in a dense architecture so that the former features, which can be lost in the forward-propagation process, can be reused. We evaluated our model with different numbers of filter channels and experimented with it using the Dop-Net dataset, with different time lengths of input data. As a result, it was found that the model with 64 3 × 3 filters and 200 time bins of micro-Doppler spectrogram data could achieve the best performance trade-off, with 99.87% classification accuracy and 3.1 ms latency. In comparison, our model remarkably outperformed the selected state-of-the-art neural networks (GoogLeNet, Res- Net-50, NasNet-Mobile, and MobileNet-V2) using the same Dop-Net dataset.

Low personality-sensitive feature learning for radar-based gesture recognition

Radar-based sensing of gestures has gained tremendous attention with the recent advancements in radar technologies. However, evident discrepancies exist in the gesture samples due to hand flexibility and individual habits. It is challenging for traditional methods to identify the gestures from unknown data sources. Cross-person (Cross-scenario) recognition refers to a recognition where the training and test samples are from different people (scenarios), respectively. To explore how the recognition performance is affected by the individual habits, the reasons are analyzed and visualized through the experiments. On this basis, HandNet is targeted proposed for the low personality-sensitive feature learning and it has two main contributions. First, a Stepped Data Augmentation (SDA) is proposed to reduce the sample interferences by non-coherent accumulating, and capture the inter-frame dependencies. Second, a Focus on Generalization loss (FoG loss) is proposed to highlight the generalized feature learning by res tricting the distances of inter-source features. Extensive experiments demonstrate that HandNet effectively reduces the classifier’s sensitivity to the personalized habits, and outperforms the existing state-of-the-art methods on the cross-person and cross-scenario gesture recognition. To the best of our knowledge, it is the first time to dedicate to addressing the radar-based gesture recognition with low personal sensitivity, which is more suitable for practical scenarios.

Highly-Optimized Radar-Based Gesture Recognition System with Depthwise Expansion Module.

The increasing integration of technology in our daily lives demands the development of more convenient human–computer interaction (HCI) methods. Most of the current hand-based HCI strategies exhibit various limitations, e.g., sensibility to variable lighting conditions and limitations on the operating environment. Further, the deployment of such systems is often not performed in resource-constrained contexts. Inspired by the MobileNetV1 deep learning network, this paper presents a novel hand gesture recognition system based on frequency-modulated continuous wave (FMCW) radar, exhibiting a higher recognition accuracy in comparison to the state-of-the-art systems. First of all, the paper introduces a method to simplify radar preprocessing while preserving the main information of the performed gestures. Then, a deep neural classifier with the novel Depthwise Expansion Module based on the depthwise separable convolutions is presented. The introduced classifier is optimized and deployed on the Coral Edge TPU board. The system defines and adopts eight different hand gestures performed by five users, offering a classification accuracy of 98.13% while operating in a low-power and resource-constrained environment.

Hand character gesture recognition based on a single millimetre‐wave radar chip

In this work, a new method for hand character gesture recognition based on a single millimetre-wave radar chip is proposed. Most current radar-based gesture recognition approaches use Doppler spectrograms or the feature extraction of Doppler spectrograms. In contrast, the authors propose using the trajectory points of gesture movements. Compared with the previous methods, the variations in the gesture movement speed, distance and direction during data acquisition have relatively little impact on the generated trajectories. The character trajectories drawn by gestures in the air output a fixed shape, and this fixed output facilitates the classification of gestures at a later stage. In this work, a single millimetre-wave radar is used to collect data and a three-dimensional fast Fourier Transform (3D-FFT) algorithm is used to obtain the gesture trajectories. For the trajectory points, a trajectory conversion image method is proposed and finally the improved Xception network proposed in this work is used for image recognition. Compared with traditional methods, the proposed method does not require large-scale data acquisition to facilitate different users. This work is concluded by recognising hand character gestures and comparing them with other commonly used classification methods to verify the gesture recognition accuracy of the proposed method.

Analysis of Edge-Optimized Deep Learning Classifiers for Radar-Based Gesture Recognition

The increasing significance of technology in daily lives led to the need for the development of convenient methods of human-computer interaction (HCI). Given that the existing HCI approaches exhibit various limitations, hand gesture recognition-based HCI may serve as a more intuitive mode of human-machine interaction in many situations. In addition, the system has to be deployable on low-power devices for applicability in broadly defined Internet of Things (IoT) and smart home solutions. Recent advances exhibit the potential of deep learning models for gesture classification, whereas they are still limited to high-performance hardware. Embedded neural network accelerators are constrained in terms of available memory, central processing unit (CPU) clock speed, graphics processing unit (GPU) performance, and a number of supported operations. The aforementioned problems are addressed in this paper by namely two approaches - simplifying the signal processing pipeline to avoid recurrent structures and efficient topological design. This paper employs an intuitive scheme allowing for the generation of the data in the compressed form from the sequence of range-Doppler images (RDI). Thus, it allows for the design of a neural classifier avoiding the usage of recurrent layers. The proposed framework has been optimized for Intel® Neural Compute Stick 2 (Intel® NCS 2), at the same time achieving promising classification accuracy of 97.57%. To confirm the robustness of the proposed algorithm, five independent persons have been involved in the algorithm testing process.

Gesture-Radar: A Dual Doppler Radar Based System for Robust Recognition and Quantitative Profiling of Human Gestures

Gesture recognition is key to enabling natural human-computer interactions. Existing approaches based on wireless sensing focus on accurate identification of arm gesture types. It remains a challenge to recognize and profile the details of arm gestures for precise interaction control. In addition, current approaches have strict positioning requirements between radars and users, making them difficult for real-world deployment. In this article, we adopt the multisensor approach and present gesture-radar-a dual Doppler radar-based gesture recognition and profiling system, which can capture subtle arm gestures with less positioning or environmental dependence. Gesture-radar uses two vertically placed Doppler radars to collect complementary sensing data of gestures, based on which cross-analysis can be performed for gesture recognition and profiling. Specifically, we first propose a two-stage classification model and enhance the signal proximity matching method by applying constraint functions to the DTW algorithm, aiming to improve the accuracy of gesture type recognition. Afterward, we establish and analyze unique features from the time-frequency spectrogram, which can be used to characterize in-depth gesture details, e.g., the angle or range of an arm movement. Experimental results show that gesture-radar achieves up to 93.5% average accuracy for gesture type recognition, and over 80% precision for profiling gesture details. This proves that the proposed approach is viable and can work in real-world environments.

Real-Time Radar-Based Gesture Detection and Recognition Built in an Edge-Computing Platform

In this paper, a real-time signal processing frame-work based on a 60 GHz frequency-modulated continuous wave (FMCW) radar system to recognize gestures is proposed. In order to improve the robustness of the radar-based gesture recognition system, the proposed framework extracts a comprehensive hand profile, including range, Doppler, azimuth and elevation, over multiple measurement-cycles and encodes them into a feature cube. Rather than feeding the range-Doppler spectrum sequence into a deep convolutional neural network (CNN) connected with recurrent neural networks, the proposed framework takes the aforementioned feature cube as input of a shallow CNN for gesture recognition to reduce the computational complexity. In addition, we develop a hand activity detection (HAD) algorithm to automatize the detection of gestures in real-time case. The proposed HAD can capture the time-stamp at which a gesture finishes and feeds the hand profile of all the relevant measurement-cycles before this time-stamp into the CNN with low latency. Since the proposed framework is able to detect and classify gestures at limited computational cost, it could be deployed in an edge-computing platform for real-time applications, whose performance is notedly inferior to a state-of-the-art personal computer. The experimental results show that the proposed framework has the capability of classifying 12 gestures in real-time with a high F1-score.

Detecting Mid-Air Gestures for Digit Writing With Radio Sensors and a CNN

In this paper, we classify digits written in mid-air using hand gestures. Impulse radio ultrawideband (IR-UWB) radar sensors are used for data acquisition, with three radar sensors placed in a triangular geometry. Conventional radar-based gesture recognition methods use whole raw data matrices or a group of features for gesture classification using convolutional neural networks (CNNs) or other machine learning algorithms. However, if the training and testing data differ in distance, orientation, hand shape, hand size, or even gesture speed or the radar setup environment, these methods become less accurate. To develop a more robust gesture recognition method, we propose not using raw data for the CNN classifier, but instead employing the hand’s mid-air trajectory for classification. The hand trajectory has a stereotypical shape for a given digit, regardless of the hand’s orientation or speed, making its classification easy and robust. Our proposed method consists of three stages: signal preprocessing, hand motion localization, and tracking and transforming the trajectory data into an image to classify it using a CNN. Our proposed method outperforms conventional approaches because it is robust to changes in orientation, distance, and hand shape and size. Moreover, this method does not require building a huge training database of digits drawn by different users in different orientations; rather, we can use training databases already available in the image processing field. Overall, the proposed mid-air handwritten digit recognition system provides a user-friendly and accurate mid-air handwriting modality that does not place restrictions on users.